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Abstract 


The achievable information rates of optical communication networks have been widely 
increased over the past four decades with the introduction and development of optical 
amplifiers, coherent detection, advanced modulation formats, and digital signal processing 
techniques. These developments promoted the revolution of optical communication sys- 
tems and the growth of Internet, towards the direction of high-capacity and long-distance 
transmissions. The performance of long-haul high-capacity optical fiber communication 
systems is significantly degraded by transmission impairments, such as chromatic disper- 
sion, polarization mode dispersion, laser phase noise and Kerr fiber nonlinearities. With the 
entire capture of the amplitude and phase of the signals using coherent optical detection, 
the powerful compensation and effective mitigation of the transmission impairments can be 
implemented using the digital signal processing in electrical domain. This becomes one of 
the most promising techniques for next-generation optical communication networks to 
achieve a performance close to the Shannon capacity limit. This chapter will focus on the 
introduction and investigation of digital signal processing employed for channel impair- 
ments compensation based on the coherent detection of optical signals, to provide a 
roadmap for the design and implementation of real-time optical fiber communication 
systems. 


Keywords: optical communications, optical networks, digital signal processing, coherent 
detection, chromatic dispersion, polarization mode dispersion, laser phase noise, fiber 
nonlinearities 
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1. Introduction 


The performance of high-capacity optical communication systems can be significantly degraded 
by fiber attenuation, chromatic dispersion (CD), polarization mode dispersion (PMD), laser 
phase noise (PN), and Kerr nonlinearities [1-10]. Using coherent detection, the powerful com- 
pensation of transmission impairments can be implemented in electrical domain. With the full 
information of the received signals, the chromatic dispersion, the polarization mode dispersion, 
the carrier phase noise, and the fiber Kerr nonlinearities can be equalized and mitigated using 
digital signal processing (DSP) [11-22]. 


Due to the high sensitivity of the receiver, coherent optical transmission was investigated 
extensively in the eighties of last century [23, 24]. However, the development of coherent 
communication has been delayed for nearly 20 years after that period [25, 26]. Coherent optical 
detection re-attracted the research interests until 2005, since the advanced modulation formats, 
i.e, m-level phase shift keying (m-PSK) and m-level quadrature amplitude modulation 
(m-QAM), can be applied [27-30]. In addition, coherent optical detection allows the electrical 
mitigation of system impairments. With the two main merits, the reborn coherent detections 
brought us the enormous potential for higher transmission speed and spectral efficiency in 
current optical fiber communication systems [31, 32]. 


With an additional local oscillator (LO) source, the sensitivity of coherent receiver reached 
the limitation of the shot-noise. Furthermore, compared to the traditional intensity modula- 
tion direct detection system, the multilevel modulation formats can be applied using the 
phase modulations, which can include more information bits in one transmitted symbol than 
before. 


Meanwhile, since the coherent demodulation is linear and all information of the received 
signals can be detected, signal processing approaches, i.e., tight spectral filtering, CD equali- 
zation, PMD compensation, laser PN estimation, and fiber nonlinearity compensation, can be 
implemented in electrical domain [33-40]. 


The typical block diagram of the coherent optical transmission system is shown in Figure 1. 
The transmitted optical signal is combined coherently with the continuous wave from the 
narrow-linewidth LO laser so that the detected optical intensity in the photodiode (PD) ends 
can be increased and the phase information of the optical signal can be obtained. The use of LO 
laser is to increase the receiver sensitivity of the detection of optical signals, and the perfor- 
mance of coherent transmission can even behave close to the Shannon limit [3, 12]. 


The development of the coherent transmission systems has stopped for more than 10 years 
due to the invention of Erbium-doped fiber amplifiers (EDFAs) [1, 2]. The coherent trans- 
mission techniques attracted the interests of investigation again around 2005, when a new 
stage of the coherent lightwave systems comes out by combining the digital signal 
processing techniques [41-46]. This type of coherent lightwave system is called as digital 
coherent communication system. In the digital coherent transmission systems, the electrical 
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signals output from the photodiodes are sampled and transformed into the discrete signals 
using high-speed analogue-to-digital convertors (ADCs), which can be further processed by 
the DSP algorithms. 


The phase locking and the polarization adjustment were the main obstacles in the traditional 
coherent lightwave systems, while they can be solved by the carrier phase estimation and the 
polarization equalization, respectively, in the digital coherent optical transmission systems 
[47-55]. Besides, the chromatic dispersion and the nonlinear effects can also be mitigated by 
using the digital signal processing techniques [56-62]. The typical structure of the DSP com- 
pensating modules in the digital coherent receiver is shown in Figure 2. 


Fiber Channel 


PD 


Figure 1. Schematic of coherent optical communication system with digital signal processing. 


Carrier 
Recovery 


| CDC/DBP | 


Figure 2. Block diagram of DSP in digital coherent receiver. 


2. Digital signal processing for compensating transmission impairments 


In this section, the chromatic dispersion compensation, polarization mode dispersion equali- 
zation, and carrier phase noise compensation are analyzed and discussed using corresponding 
DSP algorithms. 
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2.1. Chromatic dispersion compensation 


Digital filters involving the time-domain least-mean-square (TD-LMS) adaptive filter, the 
static time-domain finite impulse response (STD-FIR) filter, and the frequency-domain 
equalizers (FDEs) are investigated for CD compensation. The characters of these filters are 
analyzed based on a 28-Gbaud dual-polarization quadrature phase shift keying (DP-QPSK) 
coherent transmission system using postcompensation of dispersion. It is noted that the 
STD-FIR filter and the FDEs can also be used for the dispersion predistorted coherent com- 
munication systems. 


2.1.1. Time domain least-mean-square equalizer 


The TD-LMS filter employs an iterative algorithm that incorporates successive corrections to 
weights vector in the negative direction of the gradient vector, which eventually leads to a 
minimum mean square error [34, 38, 63-65]. The transfer function of the TD-LMS digital filter 
can be described as follows: 


Your | ) = W, tg (1) Xin (1) (1) 
Wins(n + 1) = Wıns(n) + Hims Xin (n)efms(n) (2) 
eıms(n) = dıms(n) — You (1) (3) 


where x;„(n) is the vector of received signals, y,,;(7) is the equalized output signal, n is the 


index of signal, Wins(n) is the vector of tap weights, H is the Hermitian transform operator, 
dıms(n) is the desired symbol, e,ms(n) is the error between the desired symbol and the output 
signal, * is the conjugation operator, and urns is the step size. To ensure the convergence of tap 


weights Wims(n), the step size ums has to meet the condition of Uy ms < 1/Umax, where Umax 


—_ +H 
is the largest eigenvalue of the correlation matrix R = x„(n)x,„(n) [63]. The TD-LMS disper- 
sion compensation filter can be applied in the “decision-directed” or the “sequence-training” 
mode [63]. 


The tap weights in TD-LMS adaptive equalizer for 20 km fiber CD compensation are shown in 
Figure 3. The convergence for 9 tap weights in the TD-LMS filter with step size equal to 0.1 is 
shown in Figure 3(a), and it is found that the tap weights reach their convergence after ~5000 
iterations. The distribution of the magnitudes of the converged tap weights is plotted in 
Figure 3(b), and it is found that the central tap weights take more dominant roles than the 
high-order tap weights [34, 66]. 


2.1.2. Static time-domain finite impulse response filter 


Compared with the iteratively updated TD-LMS filter, the tap weights in STD-FIR filter have a 
relatively simple specification [34, 67-69], the tap weight in STD-FIR filter is given by the 
following equations: 
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Figure 3. Taps weights of TD-LMS filter. (a) Tap weights magnitudes convergence and (b) converged tap weights 
magnitudes distribution. 
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where D is the CD coefficient, A is the carrier central wavelength, L is the length of fiber, T is 
the sampling period, N“ is the maximum number of taps, and |x] means the nearest integer 
smaller than x. 
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Figure 4. Tap weights of STD-FIR chromatic dispersion compensation filter. 


For 20 km fiber with CD coefficient of D = 16ps/(nm - km), the distribution of the tap weights 
in the STD-FIR filter is shown in Figure 4. 


2.1.3. Frequency domain equalizers 


Since the complexity is very low for compensating large CD [34, 70], the most promising and 
popular chromatic dispersion compensation filters in coherent transmission systems are the 
frequency domain equalizers. The transfer function of the frequency domain equalizers is given 
by the following expression: 


ae) (6) 


G.(L, w) = exp ( = 
where D is the chromatic dispersion coefficient, A is the carrier central wavelength, w is the 
angular frequency, L is the length of fiber, and c is the light speed in vacuum. 


The frequency domain equalizers are generally implemented using the overlap-save (OLS) and 
the overlap-add (OLA) approaches based on the fast Fourier transform and the inverse fast 
Fourier transform (iFFT) convolution algorithms [71-73], as described in Figure 5. 


2.2. Polarization mode dispersion equalization 


Due to the random character of the polarization mode dispersion and the polarization rotation, 
the compensation of the PMD and the polarization rotation are generally realized by the 
adaptive algorithms such as the least-mean-square (LMS) and the constant modulus algorithm 
(CMA) filters. 
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Frequency domain equalizer 


Figure 5. Schematic of frequency domain equalizer for chromatic dispersion compensation. 


2.2.1. LMS adaptive PMD equalization 


In the electrical domain, the impact of the PMD and the polarization fluctuation can be 
adaptively equalized using the decision-directed LMS (DD-LMS) filter [36, 63], of which the 
transfer function is given by: 
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Wry (n + 1) = Wry (Nn) + lp ` x(1) - y, (n) 
yy (M+ 1) = ln) + u, €y(m) -Y (n) 
e(n) = dy (n) — Xou (n) 
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where x;„(n) and y, (n) are the vectors of the input signals, xXou(n) and You:(n) are the equalized 


output signals, respectively, Wyx(n), Wxy(n), Wyx(n) and Wy,(n) are the complex tap weights 
vectors, d,(n) and d,(n) are the desired symbols, e,(n) and €,(n) are the estimation errors 
between the desired symbols and the output signals in the two polarizations, respectively, 
and ı1, is the step size in the DD-LMS algorithm. 


2.2.2. CMA adaptive PMD equalization 


The influence of the PMD and the polarization fluctuation can also be compensated employing 
the CMA adaptive filter [74, 75], of which the transfer function can be described as: 
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where Xin (n) and y ;, (11) are the vectors of the input signals, Xou(n) and y,,,;(1) are the equalized 


output signals, respectively, Vyx(n), Viy(n), Vyx(n), and Y(n) are the complex tap weights 
vectors, n(n) and n,(n) are the estimation errors between the desired amplitude and the output 
signals in the two polarizations, respectively, and u, is the step size in the CMA algorithm. 


It can be found that the CMA algorithm is based on the principle of minimizing the modulus 
variation of the output signal to update its weight vector. 


2.3. Carrier phase estimation 


In this section, the analyses on different carrier phase estimation algorithms, involving the one- 
tap normalized LMS, the differential phase estimation, the block-wise average (BWA), and the 
Viterbi-Viterbi (VV) methods in the coherent optical transmission systems, will be presented. 


2.3.1. The normalized LMS carrier phase estimation 
The one-tap normalized LMS filter can be employed effectively for carrier phase estimation 


[76-78], of which the tap weight is expressed as: 


I 
ST xi (n)enıms(n) (13) 


wnıms(n +1) = wnıms(n) + 


enLms(n) = dpe(n) — wnıms(N) - Xin(n) (14) 


where wynrms(n) is the tap weight, x;„(n) is the input signal, n is the symbol index, dpg(n) is the 
desired symbol, and eyzms(n) is the carrier phase estimation error between the desired symbol 
and the output signal, and unzms is the step size in the one-tap normalized LMS filter. 


It has been demonstrated that the one-tap normalized LMS carrier phase estimation behaves 
similar to the differential phase estimation [28, 53, 55, 76], of which the BER floor in the m-PSK 
coherent optical transmission systems can be approximately described by the following ana- 
lytical expression: 
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NLMS „ 
BER for ~ Ber erfc (- z) (15) 


where o is the square root of the phase noise variance. The schematic of the one-tap normalized 
LMS carrier phase estimation is illustrated in Figure 6. 


2.3.2. Differential carrier phase estimation 


The differential signal demodulation can be also applied for carrier phase estimation in coherent 
transmission system [28, 53, 55], where the differentially encoded data can be recovered using 
the “delay and multiply” algorithm. Using differential carrier phase estimation, the encoded 
information can be recovered according to the phase difference between the two consecutive 
symbols, i.e., the decision variable Y = x„x;;,, exp {it/m}, where x, and x„+ are the consecutive 
n-th and (n+1)-th received symbols. The BER floor of the differential carrier phase estimation can 
be evaluated using the principle of conditional probability. For the m-PSK coherent systems, the 
BER floor in differential phase estimation is expressed as the following equation [28, 53]: 


Differential _ 1 TC 
BER poor = Dem erfc (- =) (16) 


where o is the square root of the phase noise variance. The schematic of the differential carrier 
phase estimation is described in Figure 7. 


Xin(n ) Yy oul ) 


Figure 6. Schematic of one-tap normalized LMS carrier phase estimation. 
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Figure 7. Schematic of differential carrier phase estimation. 
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2.3.3. The block-wise average carrier phase estimation 


The block-wise average approach calculates the m-th power of the received symbols in each 
processing unit to remove the information of phase modulation, and the computed phase is 
summed and averaged over the entire process block, where the length of the process block is 
called block size. Then the averaged phase is divided by m, and the result leads to the phase 
estimate for the entire data block [79-81]. For the m-PSK coherent communication system, the 
estimated carrier phase in each process block using the block-wise average approach is given 
by the following expression: 


M-N, 


Dawa(n) = Larg 5 x” (k) (17) 
k=1+(M-1)-Np 
n 
m= | (18) 


where N, is the block size in the BWA approach, and [x] means the nearest integer larger than x. 


The performance of the block-wise average carrier phase estimation method in the m-PSK 
coherent optical communication system can be derived based on the Taylor expansion of the 
estimated carrier phase error, and the BER floor in the block-wise average carrier phase 
estimation can be described using the following expression [52, 53, 55, 79]: 


No 
BERE WA = 19 
‘floor los 2, erfc (A TOR -) ( ) 


dwak =o [2(k N +N, = 1| (20) 
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where 0? represents the total phase noise variance in the coherent transmission system. The 
schematic of the block-wise average carrier phase estimation is shown in Figure 8. 


Figure 8. Schematic of block-wise average carrier phase estimation. 
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2.3.4. The Viterbi-Viterbi carrier phase estimation 


The Viterbi-Viterbi carrier phase estimation approach also operates the symbols in each process 
block into the m-th power to remove the information of the phase modulation. The computed 
phase is also summed and averaged over the entire process block, where the length of the 
process block is also called block size. Then the averaged phase is divided by m as the estimated 
carrier phase. However, compared to the BWA approach, the estimated phase in the Viterbi- 
Viterbi carrier phase estimation approach is only applied in the phase recovery of the central 
symbol in each process block [55, 81-83]. The estimated carrier phase in the Viterbi-Viterbi 
approach in m-PSK optical communication systems is given by the following expression: 


i 1 (Ne-1)/2 
®yy(n) = —arg > x(n +k) p, No = 1,3,5,7... (21) 
m k=-(N,-1)/2 


where N, is the block size in the Viterbi-Viterbi carrier phase estimation approach. 


The performance of the Viterbi-Viterbi carrier phase estimation in the m-PSK coherent optical 
communication system can also be derived employing the Taylor expansion of the estimated 
carrier phase. The BER floor in the Viterbi-Viterbi carrier phase estimation for the m-PSK 
transmission system can be expressed as follows [52, 53, 55]: 


1 T 
BERYY = er e( ) 22 
‘floor log,m f my 2ovy ( ) 
Ne=1 
Pee Sa tO 
Oyy =o DN, (23) 


where 0? represents the total phase noise variance in the coherent transmission system. The 
schematic of the Viterbi-Viterbi carrier phase estimation is illustrated in Figure 9. 


According to Eqs. (20) and (23), it can be found that the phase estimate error in the Viterbi- 
Viterbi carrier phase estimation corresponds to the phase estimate error of the central symbol 
(the smallest error) in the block-wise average carrier phase estimation. Therefore, the Viterbi- 
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Figure 9. Schematic of Viterbi-Viterbi carrier phase estimation. 
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Viterbi approach will generally perform better than the block-wise average approach, in terms 
of the phase estimate error. However, it requires more computational complexity to update the 
process unit for the phase estimation of each symbol. 


It is noted that the one-tap normalized LMS algorithm can also be employed for the m-QAM 
coherent transmission systems, while the block-wise average and the Viterbi-Viterbi methods 
cannot be easily used for the classical m-QAM coherent systems except the circular constella- 
tion m-QAM systems. 


3. Conclusions 


In this chapter, the digital signal processing techniques for compensating transmission impair- 
ments in optical communication systems including chromatic dispersion, polarization mode 
dispersion, and laser phase noise have been described and analyzed in detail. Chromatic disper- 
sion can be compensated using the digital filters in both time domain and frequency domain. 
Polarization mode dispersion can be equalized adaptively using the least-mean-square method 
and the constant modulus algorithm. Phase noise from the laser sources can be estimated and 
compensated using the feed-forward and feed-back carrier phase recovery approaches. 


Digital signal processing combined with coherent detection shows a very promising solution 
for long-haul high-capacity optical communication systems, which offers a great flexibility in 
the design, deployment, and operation of optical communication networks. Fiber nonlinear- 
ities, including self-phase modulation, cross-phase modulation, and four-wave mixing, can be 
mitigated using single-channel and multichannel digital back-propagation in the electrical 
domain, which will be discussed in future work. 
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